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Abstract 

In this paper, we propose a new framework for compressive video sensing (CVS) that exploits the 
inherent spatial and temporal redundancies of a video sequence, effectively. The proposed method splits 
the video sequence into the key and non-key frames followed by dividing each frame into the small non¬ 
overlapping blocks of equal size. At the decoder side, the key frames are reconstructed using adaptively 
learned sparsifying (ALS) basis via fo minimization, in order to exploit the spatial redundancy. Also, 
three well-known dictionary learning algorithms are investigated in our method. For the recovery of 
non-key frames, a prediction of current frame is initialized, by using the previous reconstructed frame, in 
order to exploit the temporal redundancy. The prediction is employed into a proper optimization problem 
to recover the current non-key frame. To compare our experimental results with the results of some other 
methods, we employ peak signal to noise ratio (PSNR) and structural similarity (SSIM) index as the 
quality assessor. The numerical results show the adequacy of our proposed method in CVS. 

Index Terms 

Compressive Video Sensing, Sparse Recovery, Split Bregman Iteration, Dictionary Learning. 

I. Introduction 

Conventional approaches to image/video compression usually have high computational complexity in 
encoding hut they remain simple in the decoding process part. Also, distributed video coding (DVC) 
[1] is a low complexity technique which refers to a special video coding paradigm that encodes frames 
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of video sequence independently (the encoder can be very simple) and decodes them jointly at the 
expense of a more complex decoder (due to exploitation of the temporal redundancies by the decoder 
side). In both conventional video coding and DVC, data collection and compression tasks are performed 
disjointedly, with a high cost mechanism that wastes most valuable acquired data, because of limitation 
of allocated power and available memory. Due to great efforts by Candes et al. [2], [3] and Donoho [4], 
compressive sensing (also called compressed sensing or compressive sampling) suggests a new framework 
for simultaneous sampling and compression of signals at a rate significantly below the Nyquist rate. It 
also permits that under certain conditions, the original signal can be reconstructed properly from a small 
set of measurements via solving a convex optimization problem or iterative greedy recovery algorithms. 

Recently, the idea of compressive sensing (CS) for imaging (single pixel camera [5, 6]) has been 
extended to the conventional predictive/distributed video coding, to develop highly desirable compressive 
video sensing (CVS)/distributed compressive video sensing (DCVS). CVS employs both data acquiring 
(video sensing) and compression into a unified task which emerges a new procedure to directly acquiring 
compressed video data via random projection for each individual frame, in a low-complexity encoder. 

A. Related Works 

Several CVS recovery methods have already been proposed, i.e., Wakin et al. [7] proposed an intuitive 
(motion JPEG motivated) approach which extends compressive image sensing to video applications by 
considering each frame of the video sequence independently, and recovers each frame using the 2D 
discrete cosine transform (2D DCT) or a 2D discrete wavelet transform (2D DWT), individually. However, 
this simple extension fails to address the temporal redundancy in video; nevertheless compressed image 
sensing techniques explore the spatial redundancy within an image. To enhance the signal sparsity in 
both spatial and temporal domains and achieve higher sampling efficiency, several frames can be jointly 
considered as a signal and recovered under a 3D transform (e.g., 3D DWT) [7, 8]. Results show that a 3D 
video reconstruction (joint frames) using a 3D DWT is better than the 2D frame-by-frame reconstruction 
using a 2D DWT, but it incurs high computation cost and large memory requirement. 

Park and Wakin [9] proposed a multi-scale recovery approach, which several CS measurements are 
taken independently for each frame, and also the motion estimation is applied at the decoding step. The 
recovered video at coarse scales (low spatial resolution) is used to estimate motion which is then used to 
enhance the recovery at finer scales (high spatial resolutions). However, its coarse-to-fine reconstruction 
computational cost is high. The same approach based on using two-steps to iteratively update the estimates 
for the images in the video and the inter frame motion, was proposed in [10]. Also, Cossalter et al. [11] 
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considered the motion estimation in their proposed joint compressive video coding and analysis scheme. 

Stankovic et al. [12] and Prades-Nohet et al. [13] proposed a hlock-hased selective video sampling 
scheme where firstly divides frames of the video sequence into reference frames (or key frames) and non¬ 
reference frames (or non-key frames); then each frame is divided into the small non-overlapping blocks of 
equal size. The reference frames are sampled fully using conventional video compression techniques like 
MPEG/H.264 (intra encoding). In [12], the reference frames are used to predict sparsity of the successive 
non-reference frames and CS is applied only within the blocks that are predicted as sparse ones, whereas 
the remaining blocks are sampled fully. In [13], non-key frames are projected and recovered using CS 
techniques, with an adaptive redundant dictionary built by picking a set of local (spatially neighboring) 
blocks which are extracted from its co-located blocks in the previous decoded key frame, to form its 
basis without training. A similar approach was proposed in [14]. Nevertheless, such local dictionary-based 
basis may not work very well for blocks with large motion or when the entire scene undergoes motion 
translation. In addition such schemes highly rely on the qualities of the neighboring reconstructed key 
frames and the performance may be degraded due to poorly reconstructed neighboring key frames [15]. 

The work in [16] relies on small inter-frame differences together with a spatial 2D DWT to produce 
a sparse representation of the underlying video. Similarly, Zheng and Jakobs [17] proposed a video 
compressive sensing method that relies on the sparsity properties of video in the spatial domain, where 
key (reference) frames are fully sampled and measurements are applied to the difference between each 
pair of successive frames. The video signal is reconstructed by first reconstructing the frame differences 
using minimization algorithm, and then adding them sequentially to the reference frame. However, they 
do not sufficiently remove the vast amount of temporal redundancy, e.g., when exist a large inter-frame 
difference and fast motion between adjacent frames. 

A multi-hypothesis (MH) prediction approach for CVS was proposed in [18], where MH predictions 
of the current frame are generated from one or more previously reconstructed reference frames, and then 
combined to yield a composite prediction superior to any of the constituent single-hypothesis predictions. 

Another dictionary based approach is presented in [19], where the dictionary is learned from a set 
of blocks globally extracted from the previous reconstructed neighboring frames together with the side 
information generated from them. This work has been extended to assign dynamic measurement rate 
allocation (for different local regions) by incorporating a feedback channel [15]. Haixao et al. [20] 
proposed a redundant dictionary generation scheme for compressed video sensing, which follows the 
sparse representation approach of [13]-[14]. The authors extended their work to present a maximum- 
likelihood dictionary learning based reconstruction algorithm for DC VS [21]. Up to our best knowledge. 
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there also exist other literatures about CS-hased video coding, e.g., [22]-[23]. 

B. Contribution of This Paper 

In this paper we propose a novel for sampling and recovering of compressed sensed video data. The 
proposed method divides the video sequence into the key and non-key frames follow hy dividing each 
frame into the small blocks of equal sizes, similar to [13]. Each blocks of key (non-key) frames are 
sampled using the same sensing matrix The compressed key frame data are reconstructed 

initially using MH block compressed sensing recovery [18], in order to use as the initial image in the 
dictionary leaning algorithm step, to obtain an adaptively learned sparsifying (ALS) basis to exploit the 
spatial redundancy of frame, by an iterative procedure. Also, we investigate the effectiveness of three well- 
known dictionary learning methods, to adopt in our scheme. The obtained ALS basis is incorporated into 
the optimization problem, for the whole CS frame recovery, in form of Iq quasi-norm. In this step, a split 
Bregman iteration (SBI) [24]-[25] based technique is employed to solve the non-convex minimization, 
efficiently. For the recovery of non-key frames, first we initialize a prediction of current frame. Ft, by 
means of previous reconstructed frame Ft-i, in order to exploit the temporal redundancy. The prediction 
Ft is employed into a SBI based method together with the achieved ALS basis of current frame and 
current CS data, to recover the current non-key frame Ft by solving the proper minimization problem. 
The experimental results show the high competitive performance of our proposed method compared with 
the other state-of-the-art CVS techniques. 

The rest of this paper is organized as follows. In Section II, we introduce an abstract framework for 
CS theory and SBI method. Also, we review three well-known techniques of dictionary learning, briefly. 
The proposed method is described in Section III. Numerical results and comparisons for our proposed 
method are given in Section IV and finally. Section V concludes the paper. 

II. Background 

This section reviews an abstract framework of CS theory and SBI algorithm, accompanied by investi¬ 
gation of three well-known dictionary learning methods, briefly. 

A. Compressed Sensing 

Suppose we wish to recover a real value finite length signal u G K” from a finite length observation 
/ G M'”; so that m <C n and there is a linear projection between them 

fmxl — ^mxnUnxl + ^mxl ( 1 ) 
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where <I> G is a sensing matrix and e G denotes the additive noise. Since the number of 

unknowns is much more than the observations, clearly we are not able to recover every u from / and 
it is generally considered as an ill-posed problem. However, if u be sufficiently sparse in the sense that 
u can be written as a superposition of a small number of vectors taken from a known (sparsifying) 
transform domain basis {t = n) or frame (f > n) 'k G or even adaptive learned sparsifying 

basis (e.g., see [26]), then exact recovery of u is possible. So sparsity plays a key role in recovering 
of u from observation vector /. Also, u would be called s-sparse if only its s coefficients in the set of 
transform domain (?) = 'ku) are nonzero and the other n — s coefficients are zero. In other words, the 
transform domain signal "d can be well approximated using only s < m n nonzero entries. In order to 
solve the reconstruction problem with a reasonable accuracy and robustness to the noise, the estimation 
of u is formulated as an unconstrained Lagrangian optimization problem which incorporates the prior 
information about the original signal 

mm{^||/-+A||^'u||fJ (2) 

or 

mjn{^||/ - + A||r)||fJ, (3) 

where u = Here, the first term is a penalty that represents the closeness of the solution to the 

observed scene and quantifies the prediction error with respect to the measurements. The second term is a 
regularization term that represents a priori sparse information of the original scene and also it is designed 
to penalize an estimate that would not exhibit the expected properties. Also, A is a regularization parameter 
that balances the contribution of both terms. In the second term of (2) or (3), Ip is usually considered as 
£o or £i. This minimizing problem can be solved easily by iterative shrinkage/thresholding (1ST) methods 
(see, e.g., [27]), Bregman iterative algorithms (see, e.g., [24] and [25]). Since £o minimization is non- 
convex and its solution is considered as NP-hard, the common method is to replace Iq quasi-norm with 
the norm, because it is the closest convex norm to non-convex £q quasi-norm and minimizing the £i 
norm instead. However, a fact that is often neglected is, for some practical problems, i.e., image inverse 
problems, the conditions guaranteeing the equivalence of £q minimization and £i minimization are not 
necessarily satisfied. 

B. Split Bregman Iteration 

The split Bregman iteration (SBI) method was recently proposed by Goldstein and Osher [24] for 
effectively solving -regularized optimization problem with multiple -regularized terms [25]. The basic 
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TABLE I: The SBI ALGORITHM FRAMEWORK 


Input: {T’i(i;)},{T 2 (w)}, G, > 0 
Output: v: Reconstructed signal; 

Initialization: Set fc = 0; = (0,0,0) 

While a stop criterion is not satisfied do 

1. = argmin„ Fi(v) + ^||w* — Gv — 

2. = argmin„ F 2 {w) + ^||ui — — b'‘\\j, 

3. = 6*= + Gn''+i - 

4. fc A: + 1 
End While 
Note: w = Gv. 


idea of SBI is to convert the unconstrained minimization problem into a constrained one by introducing 
the variable splitting technique and then invoke the Bregman iteration [28] to solve the constrained 
minimization problem. The rationale behind SBI is that it may be easier to solve the constrained problem 
than to solve its unconstrained counterpart. Another advantage of the SBI is that it has relatively small 
memory footprint and is easy to program by users. Such properties are very attractive for large-scale 
problems. 

Consider an unconstrained optimization problem 

min{Fi{v) + F2{Gv)} (4) 

where G G Fi : R^ — )■ M and F2 : — )■ M. Apparently, (4) can be converted into an equivalent 

constrained form and then solved by invoking the SBI. The SBI framework is presented in Table I. 

C. Sparse Representation and Dictionary Learning 

As stated previously, the key of the sparse representation modeling lies in the choice of sparsifying 
basis (or dictionary D). In this context, D = [di,d 2 , • • • ,dt] G is called a dictionary and each of 
its columns is called an atom. One crucial problem in a sparse-representation problem is how to choose 
the efficient dictionary. There are many pre-specified (non-adaptive analytically designed) sparsifying 
dictionaries (basis or frame), e.g., Fourier, discrete cosine transform, wavelets, Ridgelets, Curvelets, 
Contourlets, Shearlets and etc. In spite of being simple and having fast computations, the analytically 
designed dictionaries are not able to efficiently (sparsely) represent a given class of signals, and they lack 
the adaptivity to the image local structures. However, learning the atoms from a set of training signals 
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belonging to signal class of interest would result in dictionaries with the capability of better matching the 
content of the signals [29]. It has been experimentally shown that these adaptive dictionaries outperform 
the non-adaptive ones in many signal processing applications. 

Dictionary learning algorithms iteratively perform the two stages of sparse approximation {sparse 
coding) and dictionary update. In the first stage, which is actually the clustering of the signals into a 
union subspace, the sparse approximation of the signals is computed using the current dictionary. The 
second stage is the update of the dictionary. To the best of our knowledge, most dictionary learning 
algorithms differ mainly in the way of updating the dictionaries [30]-[32]. Some algorithms such as 
K-singular value decomposition (K-SVD) [31] are based on updating the atoms one-by-one, while some 
others such as method of optimal directions (MOD) [30] updates the whole set of atoms at once. In [32], 
a MOD-like algorithm was proposed in which more than one atom along with the non-zero entries in 
their associated row vectors in coefficient matrix are updated at a time. We refer to this algorithm as the 
multiple dictionary update (MDU) algorithm (for further reading see [32]). 

D. Patch-based redundant sparse recovery 

In literature, the basic unit of sparse representation for natural images is patch [31]. Suppose the vector 
representation of the original image denotes by u G accordingly, Up, G represents an image patch 
of size sfWs X s/Tis at location I, I = 1,2, ■ • • , J. Then we have 

Up, = Riu, (5) 

where Ri G is a binary matrix operator that extracts the square patch ui from u, forming the output 

patch as the column vector. Note that, patches are usually overlapped (to suppress the boundary artifacts), 
and such patch based representation is highly redundant and significant to achieve high recovery quality. 
Therefore, the recovery of u from {up^} becomes an over-determind system, which is straightforward to 
obtain the following least-square solution: 

J J 

u = {Y,RjRir^Y.^Rjup,), ( 6 ) 

Z=1 1=1 

which is nothing but telling that the overall image is reconstructed by averaging all the overlapped patches. 
Given dictionary D, the sparse coding process of each patch Up^ over D is to find a sparse vector ai 
such that Up^ ss Dai. Then the entire image can be sparsely represented by the set of sparse codes {ai}: 

J J 

u^Doa = {Y^RjRi)-^Y.^RjDai), 

i=i i=i 


( 7 ) 
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where a denotes the concatenation of all {a;}. By given a set of training image patches P = [up ^, Up ^, •'' > 
where J' is the number of training image patches, the goal of sparsifying basis learning is to jointly 
optimize the sparsifying basis D, and the representation coefficient matrix A = [ai, 02 , • • • , aj']; such 
that Up, = Dai and ||a/||£^ < L, where Ip is Iq or This can be formulated by the following 
minimization problem: 

J' 

(L>, A) = argmin^ ||up, - s.t. \\ai\\i^<L,yi, (8) 

’ i=i 

where the requirement of ||ai||ro < L <C n indicates that the sparse representation stage uses no more 
than L atoms from the dictionary for every image patch instance. 

Although the above minimization problem in equation (8) is large-scale and highly non-convex even 
when Ip is equal to li, some approximation approaches (e.g., MOD [30], K-SVD [31] and MDU [32]) 
have been proposed to optimize D and A alternatively, leading to many state-of-the-art results in image 
processing. 


III. The Proposed Method 

By contrast with the conventional/distributed video coding scheme, in which data acquiring and 
compression tasks are performed disjointedly, CVS employs both data acquiring (video sensing) and 
compression, into a unified task which emerges a new procedure to directly acquiring compressed video 
data via random projection (without temporally storing the complete raw data) for each individual frame 
(or blocks of frame) at a low complexity encoder. In this case, the majority of computational burden 
is shifted from the encoder to the decoder side, which is more suitable to deploy in modern video 
applications, e.g., video serveillance systems and wireless multimedia sensor networks. In this section, 
first we discuss about encoding of the proposed method, and then we propose decoding scheme of key 
and non-key frames. 

A. Encoding 

As mentioned before, the proposed method, firstly, divides the video sequence into the key and non-key 
frames follow by dividing each frame, of size Ir x Ic, into small non-overlapping blocks of equal size 
(i.e., size B x B), and then the same sensing (sampling/measurement) matrix <h£[^(i.e., size x B^, 

*Note that, ^ denotes the measurement ratio; the measurement ratio for key and non-key frame may differ. In this context, 
MRk (MRnk) and ^bk ('f’s jvk) are used as the measurement ratio and the sensing matrix of key (non-key) frames, 
respectively. 
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where uib = = Iric) is applied for sampling of eah block. In this case, we have: 

fi = ^BUi (9) 

where m is a (column) vector representing of block i of the input image, fi is its corresponding 
measurement vector and ^b independently samples blocks within frame. Using this technique has several 
benefits comparing to use of a random sampling operator to the entire image; i.e., the encoder does not 
need to wait until the entire image is measured, but each block is sent after its linear projection. In 
addition, at the decoder side, each block is processed independently; therefore the speed of encoding 
and decoding procedure is increased. Also in this case, we just need to store a uib x sensing matrix 
instead of a m x n sensing matrix. More precisely, the global sensing matrix takes a block-diagonal 
structure, $ = diag(‘l>B, • • • , ^b). 

B. Recovery of key frame 

At the decoder side, the key frame is reconstructed initially using method of the multi-hypothesis 
block CS recovery [11], in order to use as initial training image in the process of learning an ALS basis 
(dictionary). Now, in this case, by considering u D o a. The equation (3) using ALS basis can be 
written as 

+A||a||fJ. (10) 

Here, D replaces '1'“^ in equation (3), standing for ALS basis. Also, a denotes the patch-based 
redundant sparse representation for the whole image over D, which can be find by solving (10). As 
discussed previously, when ip is replaced with io, since Iq is non-convex and NP-hard, the usual routine 
is to solve its optimal convex approximation, i.e., ii minimization. However, for some practical problems, 
i.e., image inverse problems, the conditions guaranteeing the equivalance of io minimization and ii 
minimization are not necessarily satisfied. An approach was proposed in [26], where equation (10) can 
be efficiently solved via ALS basis and fo minimization. In this part, we adopt the proposed scheme of 
[26] to solve (10), which leads to recovery of the key frames. Now, let‘s go back to equation (10) and 
point out how to solve it. By considering v = Do a and ip = iQ (sparsity is strictly measured), (10) can 
be formulated into an equivalent constrained form as: 

- ^v\\j + X\\a\\eo} s.t. v = D o a. (11) 

In order to solve the above minimization problem, an alternating SBI algorithm (as illustrated in Table. 
1) is applied. We finally obtain the following schemes: 

= argmin ^\\f — ^v\\\ + ^\\D o — v — h^\\\ 


( 12 ) 
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a 


fc+i _ 




= argminA||a ||4 + ^||-Do a - 


^k+i ^ ^ _Doot 


k+l 


, A:+1 


(13) 

(14) 


Here, // is a fixed parameter for improving the numerical stability of the algorithm. 

Given a^, the sub-problem of (12) consists in minimizing a strictly convex quadratic function, that can 
be solved easily. By setting the gradient of the objective function in (12) to be zero, a closed solution 
for (12) is achieved (see the Appendix I), which can be expressed as 

^fc+i ^ ^ /il)“^ (^(D oa^ - h^) + , (15) 

where (•)“^ and I denote the matrix inverse operator and the identity matrix, respectively. Since for 
image CS recovery and also here, <1> is a random projection matrix. Therefore, it is too costly to solve the 
minimization of the quadratic function in (12) directly by using (15), because of existence of the matrix 
inverse in (15). Here, in order to avoid computing the inversion of matrix, the steepest descent method 
with the optimal step is utilized to solve the minimization of the quadratic function in (12), which can 
be expressed as 

^k+i ^^k_ ^kgk_ 


Here, g is the gradient direction of the objective function and g represents the optimal step size; for 
finding g, see the Appendix I. 

Now, by given v in hand (according to (12)), and by considering r = v + b and u = D o a (for 
simplicity, the subscript k is dropped without confusion), the sub-problem of (13) becomes 


1„ 

min - \\u — r 
a 2 " 


li + “Il“ll4 


(17) 


By these transformations, we regard r as some type of the noisy observation of u. However, it is 
worth to note that, due to the complicated definition of a, it is difficult to solve (17) directly. In order to 
solve (17) amenable, in this paper, a reasonable assumption is used, which leads to obtain a closed-form 
solution of (17). Such this assumption is efficiently employed in [14] and [33], hence we utilize the same 
approach to solve the above minimization problem. 

Let denote the error vector by e = n — r (such that u,r G M”) and each element of e by e{i),i = 
1, • • • ,n. Also, we suppose that each element of e (e(i)) follows an independent zero-mean distribution, 
E{e{i)} = 0, with the same variance Var{e{i)} = E{e^{i)'\ = cj^, where Var{-}, E{-'\ represent the 
variance and the expectation operator, respectively. It is worth emphasizing that the above assumption 
does not need to be Gaussian, or Laplacian, or generalized Gaussian distribution process, which are 
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more general. By invoking the law of large numbers in probability theory, for any e > 0, it leads to 

lim„^oo Pr{|i XliLi ^^(0 “ ^^1 <§} = !’ 


lim Pr{| 


1 „ 

— ||ri — r\ 
n 




(18) 


where Pr{-} represents the probability. 

Let Uc, Vc denote the concatenation of all the patches Up^ and rp,, I = 1, 2, • • • , J, respectively, and 
each element of Uc — Vc is denoted by ec{j),j = 1, •'' where iP = x J. In accordance with 
the assumption, it is concluded that ec{j) is independent with zero mean and variance cr^. Thus, due 
to the law of the large numbers and by doing the same manipulation with (18) to e^(j), it yields 
limx^oo Pr{| Ejii ec(j) “ < §} = i-e-, 

J 


lim Pr{ I — 

AT—)-oo K ‘ 


1=1 




(19) 


Therefore, according to (18) and (19), the following property is concluded 

J 


lim Prjl — llu — rll?- 

1^00 ^ • r) ^ 


n-^oc "‘■2 

K—^oo l=i 

|2 _ 1 


Up, - r 


Pi Wt 


2 I < e} = 1, (20) 


in which, the relationship between ||u —r||^^ and X)i=i II^p; large probability) is described: 


J 


1 „ 

-\W-r\\f = j^2^\\up, -rp,\\f. 
1=1 


|2 _ J _ ||„. ... ||2 

K 


( 21 ) 


Now, by substituting (21) into (17), J sub-problems for all the patches Up, is achieved that they can be 
solved more efficiently. Each patch based sub-problem is formulated as 


1 


argmm-||Up, - rp,\\^^ , ik„, 

Oil l 


+ G\\(^i\ 


( 22 ) 


where 0 = ^. In fact, by considering Up, = Dai, where D is the adaptive learned dictionary from r. 


fin 


Pi 


using patch-based dictionary learning method described in Section II-D, the above sub-problem can be 
considered as the sparse coding problem, i.e.. 


argminl-WDai - rp,\\j^ + 9\\ai\\e,,. 

L 


(23) 


Also, in order to achieve higher sparsity, (23) can be formulated in its constrained form, i.e., 

i2 ^ X (24) 


min||az ||4 ^.f. \\Dai - rp,\\j < S 

Oil 


where 5 = uO is a small constant controlling the approximation error, and w is a control factor. Now, 
(24) can be solved efficiently using orthogonal matching pursuit (OMP) [34] algorithm. However, if 
£o pseudo-norm in (24) was relaxed with £i convex norm, it could be solved with basis pursuit [35], 
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TABLE II: The Detailed Description of Key Frame Recovery Framework 


Input: /, , Bs,u}, X,iin: inloop iteration number 

kmax- maximum iteration number, Tol: Tolerance 
Output: u* = D o a*: Recovered key frame; 
Initialization: Set fc = 0; (a°, 6°) = (0, 0) 

Uinit = v^= MH_recovery(/,il?B) using method of [18] 
While a stop criterion is not satisfied do 
1 . = 
rk+i ^ yk+i 

2. update using (25) 

3. for each patch Upi do 

compute using (24) 
end for 

4. update by concatenating all 

5. for i = 1 : iin 


6 . 

7. 

8 . 
9. 


gk ^ ^yk 

rji <— diag(abs( 

k+l 


fe+i 


, fe+i 


^ gf'k (>S>'k'^+llI)gk 


)) 


V 

end for 

compute s 




fe+i_ 


SSIM(t; 


fe+i 


,v) 

k\ 


compute diff = abs(s*"'"^ — s'") 
update 6* using (14) 
fc -I— fc + 1 


,,fc+i 


b’") 


end While 

stopping criterion: fc = k^ax or diff < Tol. 


lasso [36] and -regularized least square [37], but may be at tbe cost of less sparse solution. For all 
J overlapped patchs, this process is employed to achieve a (concatenation of all {«;},( = 1, • • • , J), 
which is the solution of the sub-problem (17). 

Thus, the key frame is recovered efficiently, by solving the optimization problem of (11) using discussed 
SBI method (e.g., see (12)-(14)) via updating dictionary using (8) in which, Up^ is replaced by ri, (since 
r is regarded as a good approximation of u at each iteration, here, we conduct adaptive sparsifying basis 

learning using all the patches extracted from r), 

.J' 

Z) = argmin^ ||rp, - s.t. \\ai\\(,^ < 

l=i 

The proposed algorithm for recovery of key frame data is described in detail in Table II. 


( 25 ) 
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C. Recovery of non-key frame 

While spatial domain compression is performed by CS, the temporal redundancy is not exploited 
fully, because no motion estimation and compensation is performed at the CVS encoder. To incorporate 
the temporal redundancy, in order to efficient recovery of the non-key frames, the temporal correlation 
between adjacent frames is exploited through the inter-frame sparsity model. Here, an iterative approach 
for the reconstruction of the non-key frames is adopted, where the approach initially estimates an 
approximation of the non-key frame using previous reconstructed frame, in order to take the advantage of 
the inherent temporal structure between successive frames. Then, the initially estimated frame is utilized 
into an optimization problem to recover and refine fhe currenf non-key frame. 

Assume fhaf = D o where is fhe previous reconsfructed frame using ALS basis D, 
and of_i denofes fhe pafch-based redundanf sparse represenfation for fhe whole previous reconsfructed 
frame over D. The initialization step can be formulated as: 

+^ll«lki +7'l|a - at-illti}, (26) 

a. Z 

where the first term keeps the solution close to the measurements, the second term promotes sparsity 
in the spatial transform of the current frame, and the third term promotes sparsity in the inter-frame 
difference to achieve the temporal redundancy between the current frame and the previous reconstructed 
frame. In this paper, the problem of (26) is solved efficiently by an alternating SBI-based framework via 
an iterative-shrinkage method using surrogate function. 

Assume that v = D o a, then (26) can be formulated in its constrained form: 

- ^v\\j X\\a\\e,-\-T\\a - s.t. v = D o a. (27) 

ol,v Z 

Now, the above equation can be solved easily by an alternating SBI-based framework as follow: 

= argmm {^11/ - <^v\\l + o - u - b%}, 

= argmin o a — A||a||fj -|- r||a — 

^fc+i ^ _ 2) o 

/^+i = /^ + /-$u^+i. (28) 

The last term in (28), adds a residual feedback to the algorithm. 

Given a^, b^ and /^, the first sub-problem in (28) is essentially a minimization problem of strictly 
convex quadratic function, Q 2 {v), that is 

minQ 2 (u) = - ^v\\\ -h ^\\D o -v- b^\\\}. (29) 
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Setting the gradient of the objective function in (29) to he zero, leads to a closed solution resemhle to 
that expressed in (15), hut, here / is replaced with /^. Resemhle (15), here exists matrix inverse that 
is too costly to compute, when is a random projection matrix. Therefore, again the steepest descent 
method is used to solve (29), efficiently. 

By given and assuming u = D o a and r = v + h (for the simplicity the subscript k is omitted 

without confusion), the a sub-problem in (28) can be formulated as 


1 , 


A, 


min{A||n - r\\\ + -\\a\\(,, + -\\a- 
a 2 M M 


(31) 


Now, by utilizing the same assumption which used in Section III-B, the sub-problem of (31) can be 
formulated as follow 


■ X”'' r ^ II ii2 ^*^11 II ^ w * II t 

Obviously, (32) can be solved efficiently by solving J sub-problems for all patches. By assuming u 
Dai, ^1 = ^2 = ^ and for a single patche r^,, we have 

argmin{^||T)a« - + 9i^i\\ai\\e^ +6'2,i||«« -ai-zjlti}, 


(32) 


Pi ~ 


(33) 


where 9^ = diag((6lx,z))LJ, X = {1,2}. 

The sub-problem of (33) can be solved adeptly via an iterative-shrinkage algorithm using surrogate 
function (see Appendix II, for details). Such this method is based on the work of Daubechies et al. [38], 
and was used proficiently by Dong et al. [39] in order to solve a double-header £1 optimization. 

We utilize the solution of (27) into an refinement (post processing) optimization problem in order to 
enhance the recovery of current frame. Suppose v* is the solution of (27), then the refinement optimization 
problem can be formulated as: 


{«M<}zLi,-D}= min {\\\f-^ut\\l 

at,ut,Dt 2 


1 , 

+ 2 ' 


Ut-V 


l=l 


atih 


A' 


j 

E 

i=i 


II Aaz - RiutWe.,}, 


(34) 

where fii > 0 are some regularization parameters that control the image patch sparsity. Besides, A' is 
weight parameter controls the trade-off between the data fidelity and the image prior. Here, Dt is the 
ALS basis of current frame that can be learned (initially using D and v*) and updated via the methods of 
[30]-[32]. Indeed, the second term measures the distance between the estimated frame and the enhanced 
one. The problem of (34) can be solved in an iterative mode, by decoupling it into three sub-problems of 
sparse coding, dictionary learning, and reconstruction. The sparse coding problem can be solved using 
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OMP, and Dt, in dictionary learning step, updates by the methods of [30]-[32]. Also, the reconstruction 
step has a closed-form solution 




+ I + X'Y, RlRir\v* + + A' RfDai). 


(35) 

1=1 i=i 

At a glance, it seems that (34) is very similar to that proposed in [40], however, in fact we use (34) 
in order to update the variables which used for the recovery of next frame, additionally with a minor 
enhancement on the recovered frame using (27). 


IV. Eperimental Results 

In this section, we evaluate the performance of the proposed method. To evaluate our simulation 
results, we use two applicable quality assessors, the peak signal-to-noise (PSNR) in dB and the structural 
similarity (SSIM) [41]. For an 8 bits gray scale R x R image (frame), PSNR is calculated as 

X /c255 


PSNR = 2Q\ogiQ 


(36) 


\u — u|| 

where u and u denote the original image and the reconstructed image, respectively. 

SSIM [41] is another assessor for measuring the similarity between two images, and it is near to human 
eye perception. Here, the obtained results are evaluated by a specific form of the SSIM index: 

T ci)( 2 cj UU + C2) 


SSIM{u, u) = 


(37) 


(m«^ + + Cl) (0-2 +al + C2) 

where and {ap) denote the average and variance of u (u), respectively, auu is the covariance 

of u and u. ci and C 2 are two variables to stabilize the division with weak denominator, such that 
Cl = {kiL)‘^,C 2 = {k 2 L)‘^ and 0 < ki,k 2 <C 1. T is the dynamic range of pixel values (255 for 8 
bits gray scale images). For our experiments, we use ki = 0.01 and k 2 = 0.03. All experiments were 
performed using MATFAB 2013a, on a computer equipped with Intel® core^^ i7, 3.7 GHz processor, 
with 48 GB of RAM, and running on Windows 7. The performances of our experiments are evaluated 
on the luminance component of four well-known video test sequences (e.g., “Foreman”, “Coastguard”, 
“Mobile and Calendar” and “Hall Monitor”) with a GIF resolution of 352 x 288 pixels. Also, in all 
experiments we use the block size of 32 x 32 and the size of each patch is set to 8 x 8. The CS 
measurement of each blocks is obtained by applying an orthonormalized i.i.d. Gaussian projection matrix 
to each of them, however the orthonormality of the projection matrix makes the solution of problems 
more simpler. Note that, in this paper we do not consider the quantization and entropy encoding of 
measurements, since they are beyond the scope of this paper. 
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(a) 


(b) 


(C) 


Fig. 1: Performance of different dictionary learning algorithms for the proposed method, on the frame list of Foreman sequence 


with various MRk- (a) MRk ~ 0.1: Ik-syd = 11.41 sec, Imod = 12.57 sec, tuDU = 112.33 sec. (b) MRk = 0.3: 


tK-svD = 15.1 sec, Imod = 16.05 sec, Imdu = 169.8 sec. (c) MRk = 0.5: Ik-syd = 16.96 sec, Imod = 17.78 sec. 


thiDU = 203.25 sec. 


A. Experiment 1 

In the first experiment, we evaluate the effectiveness of the discussed dictionary learning methods of 
[30]-[32] for the proposed key frame recovery algorithm. Though, the other dictionary learning methods 
can he investigated to adopt in our scheme. The key frame is reconstructed initially using the method of 
[18], in order to use as the initial training image in the process of learning an ALS basis (dictionary). 
The ALS basis is obtained by each of the methods provided in [30]-[32], in which the default parameter 
setting is as follows: the size of sparsifying basis (dictionary) is 256 and number of training iteration 
is 20. Also, in corresponding recovery problems (see Table II), iin = 200, kmax = 20, Tol = 10“^, 
p is set to be 2.5 x 10“^ and A is set empirically. In Fig. 1, a twenty-time-iteration of our method is 
illustrated as an example to show the performance of MOD [30], K-SVD [31] and MDU [32] as the 
dictionary learning methods, for recovery of the list frame of the Foreman with different MRk- In 
Fig. 1, t shows the average time of dictionary learning process at each iteration. Obviously, the results 
show that MDU provides a better recovery performance (in quality) compared to the other mentioned 
methods, but with the cost of higher computational complexity. The extra cost is only generated from 
the dictionary update step in each iteration. Also, the PSNR performance of K-SVD and MOD are very 
close to each other, but the first one gains lower computational complexity. So, it seems reasonable to 
adopt K-SVD as the dictionary learning algorithm in our method. The same experiments were run on 
Coastguard, Hall Monitor and Mobile and Calendar sequences (with fast motion scene); the obtained 
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Fig. 2: Different decodings of the list frame of Foreman with respect to recovery as a key frame (MRk ~ 0.1). (a) Original 
frame. Reconstructed 21st frame by usnig (b) The 2D DDWT basis (PSNR=26.82 dB, SSIM=0.783). (c) The multi-hypothesie 
method [11] (PSNR=28.93 dB, SS1M=0.824). The proposed method via: (d) K-SVD (PSNR=30.72 dB, SS1M=0.875). (e) MOD 
(PSNR=30.8 dB, SS1M=0.876). (f) MDU (PSNR=30.88 dB, SS1M=0.878). 


results proved the accuracy of our assumption. Fig. 2 shows the different decoding of the list frame 
of the Foreman, produced hy 2D DDWT [42] basis intra-frame decoder (with 3 level of decomposition) 
[Fig. 2(h)], the MH [18] decoder [Fig. 2(c)] and the proposed key frame recovery method using K-SVD 
[Fig. 2(d)], MOD [Fig. 2(e)] and MDU [Fig. 2(f)]. Note that, for fair comparison, the same test conditions 
(i.e., the same sensing matrix) are used in all experiments and all experimental results are averaged over 
5 independent trials. Evidently, it can he observed that the fixed basis intra-frame decoder and MH-based 
decoder suffer noticeable performance loss over the whole image, while the proposed key frame recovery 
decoder demonstrates a considerable reconstruction quality improvement. 

B. Experiment 2 

In this experiment, we evaluate the performance of the proposed method (via K-SVD dictionary 
learning) for decoding of the first 50 frames of the test video sequences, with different measurement 
ratio MR scenario. More specifically, here, two proposed CVS decoders are examined for all four test 
sequences, i.e., when = MRnk, and when MRnk < MRk- 0.5. Also, for comparision purposes, 
two existing typical CVS decoders is considered, e.g., the 2D DDWT basis intra-frame decoder, and the 
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Measurement Rate 


Measurement Rale (MR^^) 


(j) 


(k) 


Coastguard 


Coastguard 



Measurement Rate (MRjgj^) Measurement Rale (MR^^^) 


(1) (m) 

Fig. 3: The PSNR and SSIM performance of proposed method on the first 50 frames of the Foreman and the Coastguard 
sequence. Foreman: (a) PSNR vs. MR. (b) SSIM vs. MR. Coastguard: (c) PSNR vs. MR. (d) SSIM vs. MR. 


MH predictions inter-frame decoder [18]. The group of the picture (GOP) size is set to 5 and the default 
parameter setting is as well as experiment 1. The number of iterations in algorithm, kmax, is set to 6; 
it is clear that the higher number of iteration yields slightly better performance in quality of recovered 
sequence, but at the cost of higher execution times of the algorithm. It is worth emphasizing that, however 
by increasing the number of iterations, the quality of recovered frame increases, but after some iterations it 
yields an insignificant improvement, i.e., see Fig. 1(c) that PSNR values after the lOth iteration converge. 

In this experimental studies. Fig. 3 shows the PSNR and SSIM performance of the proposed method 
in various measurement ratios, compared with the other mentioned recovery algorithms for the Foreman 
and the Coastguard video sequences. The PSNR and SSIM values shown in Fig. 3, are averaged over 
all PSNR and SSIM values of the reconstructed frames. Fig. 3(a) and Fig. 3(b) show the PSNR and 
SSIM performance of the proposed method, respectively, compared with the 2D DDWT intra-frame and 
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(n) (o) (p) 



(q) (r) (s) 


Fig. 4: Different decoding of the A9th frame of Coastguard (w.r.t. recovery as a non-key frame MRk = 0.5, MRnk = 0.3). (a) 
Using the 2D DDWT basis intra-frame decoder (PSNR=23.02 dB, SSIM=0.518). (b) Using the MH method [18] (PSNR=24.38 
dB, SSIM=0.630). (c) Using the proposed method (PSNR=25.30 dB, SSIM=0.659). (d)-(f) Magnitude of error reconstruction 
for (a)-(c), respectively. 


the MH inter-frame recovery method {MRk = MRkk), in various measurement ratios. Figs. 3(a)-(h) 
are obtained for the Foreman sequence, while Figs. 3(c)-(d) show the same scenario as used in Figs. 
3(a)-(h), hut for the Coastguard sequence. As can he seen, the proposed method outperforms significantly 
in both PSNR and SSIM values, compared to the fixed basis infra-frame and MH infer-frame decoders. 
Figs. 4 and 5 show the decoding comparisions with different measurement ratios scenario for the 49th 
frame of the Coastguard, 40th frame of the Hall Monitor, and 30th frame of the Mobile and Calendar, 
respectively. The frames used in Figs. 4 and 5 are all regarded as the non-key frames. Figs. 4 and 5 
(d)-(f) depict the magnitude of error reconstruction (error map). Differences among the error maps are 
clearer with zoom-in. The error maps show the lower errors obtained by the proposed method compared 
to the other mentioned methods. Also, the averaged PSNR and SSIM values of all first 50 frames of the 
Mobile and Calendar and the Hall Monitor are listed in details in Table III. 

Obviously, the numerical results show that, the proposed method (in both states, MRk - MRkk and 
MRnk < MRk - 0.5) gains better performance, in both PSNR and SSIM, compared with the other 
mentioned methods. Also note that, by increasing MRk, the reconstruction quality will increase, but 
poor compression is its consequence. 
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(w) 


(X) 


(y) 


Fig. 5: Different decoding of the 30th frame of Mobile and Calendar (w.r.t. recovery as a non-key frame MRk = 0.4, MR^k = 


0.4). (a) Using the 2D DDWT basis intra-frame decoder (PSNR=22.02 dB, SSIM=0.645). (b) Using the MH method [18] 


(PSNR=24.1 dB, SSIM=0.749). (c) Using the proposed method (PSNR=26.6 dB, SSIM=0.863). (d)-(f) Magnitude of error 


reconstruction for (a)-(c), respectively. 


V. Conclusion 

The motivation of this paper is to propose a novel framework for CVS. In our proposed method, we 
employed the ALS basis and Iq minimization for recovering of key frames, while non-key frames are 
recovered hy firstly initialing a prediction of current non-key frame using previous reconstructed frame 
(in order to exploit the temporal redundancy), and then adopting the prediction into a proper optimization 
problem. Also, we investigated the effectiveness of three well-known dictionary learning algorithms (in 
order to learning an ALS basis), and we found out that MDU provides a better recovery performance 
(in quality) compared to the K-SVD and MOD, but at the cost of higher computational complexity. We 
found it reasonable to use the K-SVD as the dictionary learning algorithm in our proposed scheme. 
The numerical results show the adequacy of our proposed method in CVS, compared to the mentioned 
methods. 


Appendix I 

Consider the following minimization of the quadratic function: 

minQi(u) = Ta:in]-\\f - ^v\\} + ^\\Da^ - v - b^\\} (38) 

V V Z 2 
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TABLE III: PERFORMANCE COMPARISION IN PSNR (dB) AND SSIM VALUES BY DIFFERENT METHODS FOR THE Mobile 
and Calendar AND The Hal l Monitor VIDEO SEQUENCES _ 



non-key frame measurement ratio (MRiv/f): 

0.1 

0.2 

0.3 

0.4 

0.5 

Algorithm 

Mobile and Calendar (CIF) MRir=MRivK 

intra-frame 2D DDWT 

17.570 

19.389 

20.687 

21.948 

23.274 


0.383 

0.495 

0.575 

0.644 

0.703 

inter-frame MH [18] 

17.548 

20.359 

22.243 

23.9754 

26.047 


0.389 

0.567 

0.668 

0.745 

0.814 

Proposed method 

17.877 

21.181 

23.812 

26.461 

29.463 


0.435 

0.647 

0.772 

0.862 

0.924 

Algorithm 

Mobile and Calendar (CIF) MRx= 

=0.5 

intra-frame 2D DDWT 

18.727 

20.189 

21.214 

22.217 

23.274 


0.448 

0.536 

0.600 

0.655 

0.703 

inter-frame MH [18] 

19.255 

21.518 

23.001 

24.355 

26.047 


0.475 

0.618 

0.698 

0.759 

0.814 

Proposed method 

20.180 

22.875 

24.938 

27.009 

29.463 


0.534 

0.708 

0.807 

0.879 

0.924 

Algorithm 

Halil Monitor (CIF) MRx=MRjvif 

intra-frame 2D DDWT 

23.528 

28.893 

32.127 

34.722 

37.082 


0.761 

0.888 

0.927 

0.949 

0.963 

inter-frame MH [18] 

25.291 

29.608 

32.597 

34.953 

37.209 


0.814 

0.896 

0.930 

0.947 

0.961 

Proposed method 

26.133 

31.428 

35.414 

38.765 

41.379 


0.845 

0.926 

0.953 

0.970 

0.979 

Algorithm 

Hall Monitor (CIF) MRk=0.5 

intra-frame 2D DDWT 

26.235 

30.530 

33.109 

35.194 

37.082 


0.849 

0.919 

0.934 

0.952 

0.963 

inter-frame MH [18] 

27.525 

31.003 

33.419 

35.304 

37.209 


0.867 

0.921 

0.936 

0.952 

0.961 

Proposed method 

28.983 

33.268 

36.447 

39.189 

41.379 


0.890 

0.935 

0.959 

0.972 

0.979 


that can be solved by setting its gradient to be zero: 


VQi = g\ = -^^(/ - - ii{Da^ -v-b^) 

= -b^) + BV = {) 

^ + iil)v = KDa^ - b’") + 
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^v = - h^) + $^/). 

In order to avoid using matrix inverse, the gradient descent method is utilized, i.e., 

+ r?f>0. (39) 

By incorporating (39) into (38), and setting its partial derivative with respect to v/l to he zero, the optimal 
step size, r/f, is yielded, i.e., (for simplicity, the subscript k in and gf is omitted without confusion) 

= 9i {Da’^ -v + gigi - 6^) = 0, 

^ + agJvm = - ^^^) - aaliDa’^ - v- ^^), 

^ gj (‘h'^cp + Iji)r,,g, = gT (-ch^(/ - <hu) - giDa’^ -v- b’^)) 

9i 

T / 9^91 \ 

Appendix II 


The ohjective function in (33) is given hy 

f{a,at-g) = -||i9a; - rp^Wj^ + 9gi\\ai\\e^ + 92,i\\ai - 

Let us add to it the following term 

c 1 

dist(ai,ao) = -\\ai - ao\\j^ - -\\Dai - Dao\\j^, 

where the parameter c is chosen in such a way that the function dist(-, •) is strictly convex (with respect 
to ai), implying that its Hessian should he positive-definite, cI — D^D > 0. This is satisfied by the choice 
c > ||i9^T)||£2 = Amax(f9^L>) (the maximal eigenvalue of the matrix D^D). The rationale behind this 
manipulation is to convert the objective function to a new one, for which we are able to get a closed-form 
expression for its global minimizes The new objective function in (40) is called a surrogate function 

~ 1 cl 

f{a,at-i,,ao) = -\\Dai - rp^\\j^ + 9i^i\\ai\\g +92,i\\ai - + -\\ai - ao\\\ - -||i9a; - Da^W^. 

(40) 


The above function can be simplified as follow 

|2 


/(a,at_i,,ao) = \\\Dai\\\ + ^\\rp,\\j^ - afo'^rp, + 9gi\\ai\\i, 


+92,i\\oii — Ot-i, Iki + xll «/||?2 + xI|q^o ||?2 — caf ao 


-\\\Dai\\l - ^ll-^«o|li + ajD'^Dao 
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and by doing more simplification, it yields 


f{a,at-i^,ao) = consti + 9i,i\\ai\\i^ + 62 ,i\\ai - - cafvo, (41) 

where vq = \D'^{rp^ - Dao) + oq- 

As can be seen, the term drops in the new function that makes the minimization much simpler. 

Also, the term consti in (41) implying all the terms that are dependent on and ao alone. 

By simplifying and reorganizing (41), the surrogate objective function can be written as 

7/ \ _ I I II II I ^ 2 ,/ II * II I ^ II l |2 

J (cr, Oit—ii , cro) — const 2 H-||q;; H-||Q^Z “ ii lui 4" 9 ll'^z ~ ’^o\\£ 2 - 

c c ^ 

The minimizer of the above function can be obtained by an iterative-shrinkage procedure, producing 
sequence of results {a^ (the subscript i denotes the ith entry in ai, and the subscript k' implies the 
iteration number), where at the k + 1 th iteration, / (and of course /) is minimized with the assignment 
ao = af as follow 


k'+l _ 


sgn(A 




(sgn(at+i^ )vf), 


where 


^; = —,x = {i,2}, 


V = i'^Pi - Dal )+af , 


and the generalized shrinkage operator defines as follow 


'Se[,92,p{x) 


X + + @2) 

0 , 

< X — 9[ + 02, 
p, 

x-9[- 9'2 


X < —9l — 6*2 
-91 -9'2<x <9[-9'2 
9'i — 62 < X < 9l — 02 + p 
9[ — 02 + p < X < 9[ + 02 + p 
X 1> 0^ 02 P 
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